In [1]:
import os
import pandas as pd
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns
from IPython.display import display
import math
import plotly.offline as py
from plotly.offline import init_notebook_mode, iplot
import plotly.graph_objs as go
import plotly.offline as offline
import cufflinks as cf
py.init_notebook_mode(connected=True)
init_notebook_mode(connected=True)
offline.init_notebook_mode()
cf.go_offline()
color = sns.color_palette()
In [2]:
heloc = pd.read_excel("heloc_dataset_v1.xlsx")
display(heloc.tail(n=5).transpose())
Balance of input datastet¶
In [3]:
# Investigate balance for both output classes
n_records = heloc.shape[0]
n_riskperf_good = heloc.loc[heloc['RiskPerformance'] == 'Good'].shape[0]
n_riskperf_bad = heloc.loc[heloc['RiskPerformance'] == 'Bad'].shape[0]
good_percent = (n_riskperf_good/n_records)*100
print("Total number of records: {:0,}".format(n_records))
print("Individuals with good RiskPerformance: {:0,}".format(n_riskperf_good))
print("Individuals with bad RiskPerformance: {:0,}".format(n_riskperf_bad))
print("Percentage of individuals with good RiskPerformance: {:0.2f}%".format(good_percent))
In [4]:
temp = heloc["RiskPerformance"].value_counts()
df = pd.DataFrame({'labels': temp.index,
'values': temp.values
})
print(df)
df.iplot(kind='pie',labels='labels',values='values', title='RiskPerformance')
Featureset Exploration¶
FICO uses 5 broad categories of information, each assigned with a relative importance, to determine credit score. The features provided in the input dataset have been subjectively mapped to these categories below (relative weight in bracket) in order to develop a better intuition of the input features.
- Payment History (35%) : Considers payment history from different types of accounts, public record and collection items and details on late or missed payments.
- NumSatisfactoryTrades: Number of credit agreements with on-time payments
- NumTrades60Ever2DerogPubRec: Number of trade lines on a credit bureau report that record a payment received 60 days past its due date. This feature also checks all Public Records available for the consumer, and adds to this count any items considered “Derogatory”.
- NumTrades90Ever2DerogPubRec: Number of trade lines on a credit bureau report that record a payment received 90 days past its due date. This feature also checks all Public Records available for the consumer, and adds to this count any items considered “Derogatory”.
- PercentTradesNeverDelq: Percent Trades Never Delinquent
- MSinceMostRecentDelq: Months Since Most Recent Delinquency
- MaxDelq2PublicRecLast12M: Categorical variable denoting how severe was delinquency in last 12 months, if at all and known
- MaxDelqEver: Categorical variable denoting how severe was delinquency in the past, if at all and known
- Amount of Debt (30%) : Considers total amount owed across all accounts, amounts owed on specific types (revolving, credit) of accounts, number of accounts with balance, credit utilization ratio on revolving accounts, remaining amount owed on instalment loans
- NetFractionRevolvingBurden: Revolving balance divided by credit limit
- NetFractionInstallBurden: Installment balance divided by original loan amount
- NumBank2NatlTradesWHighUtilization: Number of credit cards carrying a balance at 75% of its limit or greater
- PercentTradesWBalance: Percent Trades with Balance
- Length of Credit History (15%) : Considers age of oldest account, average account age and age of specific types of accounts (credit card, auto loans, etc)
- MSinceOldestTradeOpen: Months Since Oldest Trade Open
- AverageMInFile: Average history length of trades in months
- New Credit (10%) : Considers the number of new accounts, how long since new account opened, number of recent requests for credit and rate shopping for a single loan
- MSinceMostRecentTradeOpen: Months Since Most Recent Trade Open
- NumTradesOpeninLast12M: Number of Trades Open in Last 12 Months
- MSinceMostRecentInqexcl7days: Months Since Most Recent Inquiry excl 7days
- NumInqLast6M: Number of Inquiry Last 6 Months
- NumInqLast6Mexcl7days: Number of Inq Last 6 Months excl 7days (to account for price comparison shopping)
- Credit Mix (10%) : Considers types of credit accounts (credit cards, retail accounts, instalment loans and mortgage loans)
- PercentInstallTrades: Percent Installment Trades
- NumRevolvingTradesWBalance: Number Revolving Trades with Balance
- NumInstallTradesWBalance: Number Installment Trades with Balance
- Other Category : Remaining features which could not be mapped to one of the above FICO categories
- ExternalRiskEstimate: Consolidated risk estimation from other credit bureaus
- NumTotalTrades: Number of Total Trades (total number of credit accounts)
In [5]:
display(heloc.describe().transpose())
Investigation of Special Values¶
In [6]:
print("{:30s} {:} {:} {:}".format("Feature", "No Record/Invest.", "No Usable/Valid Trade/Inq", "Cond. not met" ))
for col in heloc.columns:
no_record = heloc.loc[heloc[col] == -9].shape[0]
no_validtrades = heloc.loc[heloc[col] == -8].shape[0]
no_inq_delq = heloc.loc[heloc[col] == -7].shape[0]
print("{:30s} \t {:0,} \t\t\t {:0,} \t\t\t {:0,}".format(col[:30], no_record, no_validtrades , no_inq_delq ))
There are 3 special values in the HELOC dataset:
- No Bureau record or No Investigation (entry value =-9): There are at least 588 such entries detected for every feature provided in the dataset (excluding the target variable). If these entries belong to the same set of customers, the entries will be dropped as they offer no usable features to train the learning algorithm.
- No Usable/Valid trades or Inquiries (entry value =-8): The NetFractionInstallBurden feature displays a large gap in terms of usable/valid trades. Once the importance of this feature is determined in predicting RiskPerformance, this particular gap might need to be revisited.
- Conditions not met (e.g No Inquiries, no delinquencies) (entry value =-7): The MSinceMostRecentDelq feature has 4.6K entries with this special value, reflecting no previous delinquency. This might be a very useful piece of information in determining RiskPerformance.
Applicants with no bureau record or investigation¶
According to the data provider, when a loan applicant’s credit bureau report was either not investigated or not found, all features obtained from the credit bureau report receive a special value of -9. The confounding of no bureau report investigated (most likely a VIP applicant) and no bureau report found (a negative trait for extending credit) is most likely responsible for the two different outcomes.
All such records will be removed from the dataset due to their confounding nature.
In [7]:
no_rec= (heloc.loc[heloc['MSinceOldestTradeOpen'] == -9])
#display(no_rec)
display(no_rec.describe().transpose())
In [8]:
#Remove 588 entries with no bureau records
clean_norec_heloc = heloc.copy()
clean_norec_heloc.drop(clean_norec_heloc[clean_norec_heloc.MSinceOldestTradeOpen == -9].index, inplace=True)
In [9]:
n_unknown_DelqLast12M= (clean_norec_heloc.loc[clean_norec_heloc['MaxDelq2PublicRecLast12M'] > 7]).shape[0]
n_unknown_DelqEver_1= (clean_norec_heloc.loc[clean_norec_heloc['MaxDelqEver'] == 1]).shape[0]
n_unknown_DelqEver_7= (clean_norec_heloc.loc[clean_norec_heloc['MaxDelqEver'] == 7]).shape[0]
n_unknown_DelqEver_9= (clean_norec_heloc.loc[clean_norec_heloc['MaxDelqEver'] == 9]).shape[0]
print("Individuals with unusable delinquency data over last 12 months: {:0,}".format(n_unknown_DelqLast12M))
print("Individuals with no delinquency data: {:0,}".format(n_unknown_DelqEver_1))
print("Individuals with unknown delinquency: {:0,}".format(n_unknown_DelqEver_7))
print("Individuals with unusable delinquency data: {:0,}".format(n_unknown_DelqEver_9))
In [10]:
display(clean_norec_heloc.loc[clean_norec_heloc['MaxDelq2PublicRecLast12M'] > 7].transpose())
In [11]:
clean_nodelq_heloc = clean_norec_heloc.copy()
clean_nodelq_heloc.drop(clean_nodelq_heloc[clean_nodelq_heloc.MaxDelq2PublicRecLast12M > 7].index, inplace=True)
display(clean_nodelq_heloc.describe().transpose())
Feature Engineering¶
Feature Correlation¶
Based on the correlation heatmap below, the following features exhibit strong correlation:
- NumInqLast6M and NumInqLast6Mexcl7days : NumInqLast6M will be dropped from the feature set as NumInqLast6Mexcl7days accounts for price shopping.
- NumTrades60Ever2DerogPubRec and NumTrades90Ever2DerogPubRec : NumTrades90Ever2DerogPubRec will be dropped from the feature set as it is a subset of NumTrades60Ever2DerogPubRec
- NumSatisfactoryTrades and NumTotalTrades: NumTotalTrades will be dropped after it has been used to convert features that are expressed as a % of number of trades (PercentTradesNeverDelq, PercentInstallTrades, PercentTradesWBalance)
In [12]:
from seaborn import heatmap
# Pretty display for notebooks
%matplotlib inline
plt.figure(figsize=(25, 25))
heatmap(clean_nodelq_heloc.iloc[:, 1:].corr(),annot= True, square= True, cmap="RdYlGn")
Out[12]:
In [13]:
#clean_nodelq_heloc.drop(['NumInqLast6M','NumTrades90Ever2DerogPubRec'],axis=1, inplace=True)
clean_nodelq_heloc['NumTradesNeverDelq'] = np.rint(clean_nodelq_heloc['PercentTradesNeverDelq']* clean_nodelq_heloc['NumTotalTrades']/100)
clean_nodelq_heloc['NumInstallTrades'] = np.rint(clean_nodelq_heloc['PercentInstallTrades']* clean_nodelq_heloc['NumTotalTrades']/100)
clean_nodelq_heloc['NumTradesWBalance'] = np.rint(clean_nodelq_heloc['PercentTradesWBalance']* clean_nodelq_heloc['NumTotalTrades']/100)
display(clean_nodelq_heloc.head().transpose())
Transforming Categorical Features¶
The features MaxDelq2PublicRecLast12M and MaxDelqEver will have their values grouped in the following categories:
- Derogatory
- Delinquent (30, 60, 90, 120+ days)
- Never delinquent (current)
- Invalid (none of the above categories)
These categories will be one-hot encoded to add them as distinct features within the dataset.
In [14]:
delq12M_map = {1: 'DelinquentLast12M',
2: 'DelinquentLast12M',
3: 'DelinquentLast12M',
4: 'DelinquentLast12M',
5: 'InvalidDelqLast12M',
6: 'InvalidDelqLast12M',
7: 'NotDelinquentLast12M',
8: 'InvalidDelqLast12M',
9: 'InvalidDelqLast12M',
0: 'DerogatoryLast12M'}
delqEver_map = {1: 'InvalidDelq',
2: 'Derogatory',
3: 'Delinquent',
4: 'Delinquent',
5: 'Delinquent',
6: 'Delinquent',
7: 'InvalidDelq',
8: 'NeverDelinquent',
9: 'InvalidDelq',
}
clean_nodelq_heloc['MaxDelq2PublicRecLast12MLabel'] = clean_nodelq_heloc['MaxDelq2PublicRecLast12M'].map(delq12M_map)
clean_nodelq_heloc['MaxDelqEverLabel'] = clean_nodelq_heloc['MaxDelqEver'].map(delqEver_map)
full_features = pd.get_dummies(clean_nodelq_heloc,columns=["MaxDelq2PublicRecLast12MLabel","MaxDelqEverLabel"])
encoded = list(full_features.columns)
print("{} total features after one-hot encoding. \n".format(len(encoded)))
display(full_features.head().transpose())
In [15]:
plt.figure(figsize=(25, 25))
heatmap(full_features.iloc[:, 1:].corr(),annot= True, square= True, cmap="RdYlGn")
Out[15]:
Removal of features exhibiting collinearity¶
The feature pairs exhibiting absolute correlation greater than 0.80 (in descending order) are listed below:
- NumInqLast6M and NumInqLast6Mexcl7days : NumInqLast6M will be dropped from the feature set
- NumTotalTrades and NumTradesNeverDelq : NumTotalTrades will be dropped from the feature set
- MaxDelqEverLabel_NeverDelinquent and MaxDelq2PublicRecLast12MLabel_NotDelinquentLast12M : MaxDelq2PublicRecLast12MLabel_NotDelinquentLast12M will be dropped from the feature set
- NumTrades60Ever2DerogPubRec and NumTrades90Ever2DerogPubRec : NumTrades90Ever2DerogPubRec will be dropped from the feature set
- NumSatisfactoryTrades and NumTradesNeverDelq: NumTradesNeverDelq will be dropped from the feature set
- MaxDelqEverLabel_Delinquent and MaxDelqEverLabel_NeverDelinquent : MaxDelqEverLabel_Delinquent will be dropped
In addition, the following features will be dropped as they have already been used to derive other features:
- PercentTradesNeverDelq
- PercentTradesWBalance
- PercentInstallTrades
- MaxDelq2PublicRecLast12M
- MaxDelqEver
- MaxDelq2PublicRecLast12MLabel_InvalidDelqLast12M
- MaxDelqEverLabel_InvalidDelq
In [16]:
features_drop =['NumInqLast6M','NumTotalTrades','MaxDelq2PublicRecLast12MLabel_NotDelinquentLast12M',
'NumTrades90Ever2DerogPubRec','NumTradesNeverDelq','MaxDelqEverLabel_Delinquent'
,'PercentTradesNeverDelq','PercentTradesWBalance','PercentInstallTrades',
'MaxDelq2PublicRecLast12M','MaxDelqEver','MaxDelq2PublicRecLast12MLabel_InvalidDelqLast12M',
'MaxDelqEverLabel_InvalidDelq']
reduced_features = full_features.copy()
reduced_features.drop(features_drop,axis=1, inplace=True)
num_features = list(reduced_features.columns)
print("{} Number of features after reduction \n".format(len(num_features)))
display(reduced_features.head().transpose())
In [17]:
plt.figure(figsize=(25, 25))
heatmap(reduced_features.iloc[:, 1:].corr(),annot= True, square= True, cmap="RdYlGn", linewidths=.05)
Out[17]:
Feature Visualisation¶
In [18]:
#Produce KDEplot for a specific feature - visualise distribution of continuous numeric feature
def kdeplt(feature, rnd =-1):
xlow= round(reduced_features[feature].min(),-1)
xhigh =round(reduced_features[feature].max(),rnd)
print('Low boundary : ', xlow,'High boundary : ', xhigh)
plt.figure(figsize=(7,4))
ax = sns.kdeplot(reduced_features[feature][reduced_features['RiskPerformance'] == 'Good'], color="darkturquoise", shade=True)
sns.kdeplot(reduced_features[feature][reduced_features['RiskPerformance'] == 'Bad'], color="lightcoral", shade=True)
plt.legend(['Good', 'Bad'])
plt.title('Loan types by '+feature)
ax.set(xlabel=feature)
plt.xlim(xlow,xhigh)
plt.show()
#Multi-bar Plot - discrete categorical attributes
def barplot(feature):
plt.figure(figsize=(4,4))
cp = sns.countplot(x=feature, hue="RiskPerformance", data=reduced_features)
# Using multiple Histograms - visualising numeric and categorical attributes together
def multhist(feature):
fig = plt.figure(figsize = (6, 4))
title = fig.suptitle('Loan types by '+feature)
#fig.subplots_adjust(top=0.85, wspace=0.3)
ax = fig.add_subplot(1,1, 1)
ax.set_xlabel("MSinceOldestTradeOpen")
ax.set_ylabel("Count")
g = sns.FacetGrid(reduced_features, hue='RiskPerformance')
g.map(sns.distplot, feature, kde=False, bins=20, ax=ax)
ax.legend(title='RiskPerformance')
plt.close(2)
ExternalRiskEstimate¶
In [19]:
reduced_features['ExternalRiskEstimate'].describe()
Out[19]:
In [20]:
kdeplt('ExternalRiskEstimate',-2)
MSinceOldestTradeOpen¶
In [21]:
reduced_features['MSinceOldestTradeOpen'].describe()
Out[21]:
In [22]:
kdeplt('MSinceOldestTradeOpen',-3)
MSinceMostRecentTradeOpen¶
In [23]:
reduced_features['MSinceMostRecentTradeOpen'].describe()
Out[23]:
In [24]:
multhist('MSinceMostRecentTradeOpen')
AverageMInFile¶
In [25]:
reduced_features['AverageMInFile'].describe()
Out[25]:
In [26]:
kdeplt('AverageMInFile')
NumSatisfactoryTrades¶
In [27]:
reduced_features['NumSatisfactoryTrades'].describe()
Out[27]:
In [28]:
kdeplt('NumSatisfactoryTrades')
NumTrades60Ever2DerogPubRec¶
In [29]:
reduced_features['NumTrades60Ever2DerogPubRec'].describe()
Out[29]:
In [30]:
multhist('NumTrades60Ever2DerogPubRec')
MSinceMostRecentDelq¶
In [31]:
reduced_features['MSinceMostRecentDelq'].describe()
Out[31]:
In [32]:
multhist('MSinceMostRecentDelq')
NumTradesOpeninLast12M¶
In [33]:
reduced_features['NumTradesOpeninLast12M'].describe()
Out[33]:
In [34]:
kdeplt('NumTradesOpeninLast12M')
MSinceMostRecentInqexcl7days¶
In [35]:
reduced_features['MSinceMostRecentInqexcl7days'].describe()
Out[35]:
In [36]:
multhist('MSinceMostRecentInqexcl7days')
NumInqLast6Mexcl7days¶
In [37]:
reduced_features['NumInqLast6Mexcl7days'].describe()
Out[37]:
In [38]:
multhist('NumInqLast6Mexcl7days')
NetFractionRevolvingBurden¶
In [39]:
reduced_features['NetFractionRevolvingBurden'].describe()
Out[39]:
In [40]:
kdeplt('NetFractionRevolvingBurden')
NetFractionInstallBurden¶
In [41]:
reduced_features['NetFractionInstallBurden'].describe()
Out[41]:
In [42]:
kdeplt('NetFractionInstallBurden')
NumRevolvingTradesWBalance¶
In [43]:
reduced_features['NumRevolvingTradesWBalance'].describe()
Out[43]:
In [44]:
kdeplt('NumRevolvingTradesWBalance')
NumInstallTradesWBalance¶
In [45]:
reduced_features['NumInstallTradesWBalance'].describe()
Out[45]:
In [46]:
multhist('NumInstallTradesWBalance')
NumBank2NatlTradesWHighUtilization¶
In [47]:
reduced_features['NumBank2NatlTradesWHighUtilization'].describe()
Out[47]:
In [48]:
multhist('NumBank2NatlTradesWHighUtilization')
NumInstallTrades¶
In [49]:
reduced_features['NumInstallTrades'].describe()
Out[49]:
In [50]:
multhist('NumInstallTrades')
NumTradesWBalance¶
In [51]:
reduced_features['NumTradesWBalance'].describe()
Out[51]:
In [52]:
multhist('NumTradesWBalance')
MaxDelq2PublicRecLast12MLabel_DelinquentLast12M¶
In [53]:
reduced_features['MaxDelq2PublicRecLast12MLabel_DelinquentLast12M'].value_counts()
Out[53]:
In [54]:
barplot('MaxDelq2PublicRecLast12MLabel_DelinquentLast12M')
MaxDelq2PublicRecLast12MLabel_DerogatoryLast12M¶
In [55]:
reduced_features['MaxDelq2PublicRecLast12MLabel_DerogatoryLast12M'].value_counts()
Out[55]:
In [56]:
barplot('MaxDelq2PublicRecLast12MLabel_DerogatoryLast12M')
MaxDelqEverLabel_Derogatory¶
In [57]:
reduced_features['MaxDelqEverLabel_Derogatory'].value_counts()
Out[57]:
In [58]:
barplot('MaxDelqEverLabel_Derogatory')
MaxDelqEverLabel_NeverDelinquent¶
In [59]:
reduced_features['MaxDelqEverLabel_NeverDelinquent'].value_counts()
Out[59]:
In [60]:
barplot('MaxDelqEverLabel_NeverDelinquent')
Feature Selection¶
Principal Component Analysis¶
The first 2 principal components explain almost 85% of the variance in the data. The key features of the first three components are listed below:
- First Component (72.5% variance): The two most prominent features are MSinceOldestTradeOpen and AverageMInFile. This component captures the length of the credit history of the client.
- Second Component (12.1% variance): NetFractionInstallBurden and NetFractionRevolvingBurden are the main contributors to this component together with a negative contribution from AverageMInFile. This is intuitive as the debt burden increases monotically with RiskPerformance while AverageMInFile decreases monotically with RiskPerformance. This component captures the amount of debt of the client accounting for the length of his/her credit history.
In [63]:
from sklearn.decomposition import PCA
import visuals as vs
pca = PCA(n_components=3)
pca.fit(reduced_features.iloc[:, 1:])
# Generate PCA results plot
pca_results = vs.pca_results(reduced_features.iloc[:, 1:], pca,24,20)
Recursive Feature Selection¶
The maximum score of 71.6% is obtained when 18 features are used. From a dimensionality reduction perspective, 13 features provide a reasonable tradeoff between score (71.5%) and number of features. The 8 selected features can be grouped into 4 categories:
- Payment History : NumSatisfactoryTrades, NumTrades60Ever2DerogPubRec, MaxDelq2PublicRecLast12MLabel_DerogatoryLast12M and MaxDelqEverLabel_NeverDelinquent all relate to the previous payment history of the borrower.
- New Credit : MSinceMostRecentInqexcl7days and NumInqLast6Mexcl7days reflect how active the borrower has been in seeking new loans.
- Amount of Debt : NumRevolvingTradesWBalance is the number of active credit accounts held by the borrower.
- ExternalRiskEstimate : This is the least transparent of all the features since it is the credit score assigned to the borrower by other credit bureaus based on their assessment of the creditworthiness of the borrower.
When ExternalRiskEstimate is dropped from the list of features and the recursive feature selection is rerun, the following 13 features result in a score of 71.5%:
- Payment History : NumSatisfactoryTrades, NumTrades60Ever2DerogPubRec, MaxDelq2PublicRecLast12MLabel_DelinquentLast12M, MaxDelq2PublicRecLast12MLabel_DerogatoryLast12M, MaxDelqEverLabel_Derogatory and MaxDelqEverLabel_NeverDelinquent all relate to the payment history of the borrower.
- New Credit : NumTradesOpeninLast12M, MSinceMostRecentInqexcl7days and NumInqLast6Mexcl7days reflect the recent credit activity of the borrower.
- Amount of Debt : NetFractionRevolvingBurden, NumRevolvingTradesWBalance and NumBank2NatlTradesWHighUtilization reflect to the amount of borrowing done relative to credit limits.
- Length of credit history : AverageMInFile is the average duration of the credit accounts held by the borrower.
In [64]:
reduced_features = reduced_features.reset_index(drop=True)
riskperf_raw = reduced_features['RiskPerformance']
features = reduced_features.drop('RiskPerformance' , axis = 1)
dic={'Good':0, 'Bad':1}
riskperf = riskperf_raw.map(dic)
display(riskperf.head(n=5))
In [65]:
from gmean_score import Gscore
def scre(y_true, y_pred):
scre = Gscore(y_true, y_pred)
return scre.g_mean()
In [66]:
import gmean_score as gm
from sklearn.linear_model import LogisticRegression
from sklearn.feature_selection import RFE
features1 = features.copy()
feat_drop =['ExternalRiskEstimate']
features1.drop(feat_drop,axis=1, inplace=True)
#List to store computed scores
scr=[]
model = LogisticRegression()
# RFE model for different number of features
for i in range(1,23):
print("Number of features: ",i)
rfe = RFE(model, i)
rfe = rfe.fit(features1, riskperf)
# summarize the selection of the attributes
print('Selected features: %s' % list(features1.columns[rfe.support_]))
print('Score: %f' % rfe.score(features1, riskperf))
scr.append(scre(riskperf, rfe.predict(features1)))
print('G-mean: %f' % scr[i-1])
# Plot number of features VS. G_mean scores
plt.figure(figsize=(10,6))
plt.xticks(np.arange(0, 25, 1))
plt.xlabel("Number of features selected")
plt.ylabel("G_Mean score")
plt.plot(range(1, len(scr) + 1), scr)
plt.show()
Recursive Feature Selection with Cross Validation¶
Even with cross-validation of the dataset, the results are very similar to the recursive feature elimination (without cv). The highest score is obtained when using 18 features.
In [67]:
from sklearn.feature_selection import RFECV
from sklearn.metrics import make_scorer
scoring_fnc= make_scorer(scre)
rfecv = RFECV(estimator=LogisticRegression(), step=1, cv=10, scoring=scoring_fnc)
rfecv.fit(features1, riskperf)
print("Optimal number of features: %d" % rfecv.n_features_)
print('Selected features: %s' % list(features1.columns[rfecv.support_]))
# Plot number of features VS. cross-validation scores
plt.figure(figsize=(10,6))
plt.xticks(np.arange(0, 25, 1))
plt.xlabel("Number of features selected")
plt.ylabel("Cross validation Gmean score")
plt.plot(range(1, len(rfecv.grid_scores_) + 1), rfecv.grid_scores_)
plt.show()
Tree-Based Feature Selection¶
Using a decision tree classifer, an almost perfect G-mean score (99.5%) is achieved for a tree depth of 25 levels, both with and without the feature ExternalRiskEstimate. While the large tree depth is not ideal for interpretability, it was still a useful exercise to identify the most important features resulting in the very high predictive power. The most important features, excluding ExternalRiskEstimate due to its lack of transparency, are grouped in the following categories:
- Amount of Debt : NetFractionRevolvingBurden and NumTradesWBalance reflect the various loans taken by the borrow.
- Length of Credit History : MSinceOldestTradeOpen and AverageMInFile reflect the length of the borrower's credit history.
- Payment History : NumSatisfactoryTrades, NumTrades60Ever2DerogPubRec and MSinceMostRecentDelq reflect the payment history of the borrower.
- New Credit : MSinceMostRecentInqexcl7days is the number of months since the borrower last made an inquiry for a new loan.
- Credit Mix : NumInstallTrades is the number of instalment loans held by the borrower.
In [68]:
from sklearn.tree import DecisionTreeClassifier
for i in range(1,26,4):
print('\nTree depth: ', i)
classifier = DecisionTreeClassifier(max_depth=i)
classifier.fit(features, riskperf)
print('G-mean score is: ', scre(riskperf, classifier.predict(features)))
print('Feature importance:')
for name, importance in zip(features.columns, classifier.feature_importances_):
if importance>0.05:
print(name, importance)
ExternalRiskEstimate is the most important feature for the varying tree depths. However, since it is not very transparent how this feature is calculated by other credit bureaus, the decision tree classifier will be rerun excluding this feature to assess the impact on the Gmean score.
In [70]:
for i in range(1,26,4):
print('\nTree depth: ', i)
classifier1 = DecisionTreeClassifier(max_depth=i)
classifier1.fit(features1, riskperf)
print('G-mean score is: ', scre(riskperf, classifier1.predict(features1)))
print('Feature importance:')
for name, importance in zip(features1.columns, classifier1.feature_importances_):
if importance>0.05:
print(name, importance)
Final Feature Selection¶
Using PCA, RFE and tree feature selection, the following 9 features consistently came up as important and 8 of them (ExternalRiskEstimate) were included in the final list of features for model training.
- Length of Credit History :
- MSinceOldestTradeOpen
- AverageMInFile
- Amount of debt :
- NetFractionRevolvingBurden
- NetFractionInstallBurden - No imputation will be done for the 3415 records where there is no data on the burden.
- NumBank2NatlTradesWHighUtilization
- Payment history :
- NumSatisfactoryTrades
- NumTrades60Ever2DerogPubRec
- New Credit :
- MSinceMostRecentInqexcl7days
- ExternalRiskEstimate
In [71]:
final_features =['RiskPerformance','MSinceOldestTradeOpen','AverageMInFile','NetFractionRevolvingBurden','NetFractionInstallBurden','NumBank2NatlTradesWHighUtilization','NumSatisfactoryTrades','NumTrades60Ever2DerogPubRec','MSinceMostRecentInqexcl7days']
heloc_final = reduced_features[final_features]
display(heloc_final.head())
In [72]:
heloc_final.to_excel('heloc_featureset_final.xlsx',index= False)